fix: enforce null-key rejection and mapKeyDedupPolicy in native map construction - #5854

…onstruction

`map_from_arrays` and `map_from_entries` built their maps without the entry
checks Spark's `ArrayBasedMapBuilder` performs, so a `NULL` key inside the keys
array produced a map with a `NULL` key instead of raising `NULL_MAP_KEY`, and
`spark.sql.mapKeyDedupPolicy=LAST_WIN` fell the whole expression back to Spark.

DataFusion 55 added `datafusion.spark.map_key_dedup_policy` and taught the
`datafusion-spark` map kernels to follow it, which is the missing half. Forward
Spark's `spark.sql.mapKeyDedupPolicy` to it across JNI, and pass the session's
`ConfigOptions` into `ScalarFunctionExpr` so a kernel that reads a setting sees
the session's value rather than DataFusion's defaults.

New `SparkMapFromArrays` / `SparkMapFromEntries` / `SparkStrToMap` wrappers add
the checks the upstream kernels do not perform and restate their errors as the
Spark error classes `SparkErrorConverter` turns back into `QueryExecutionErrors`:
a `NULL` key raises `NULL_MAP_KEY` ahead of any duplicate-key check, key and
value arrays of different lengths raise `MAP_KEY_VALUE_DIFF_SIZES`, and a
duplicate key under `EXCEPTION` raises `DUPLICATED_MAP_KEY` naming the key.
`CometMapFromArrays` now emits `map_from_arrays`, which is null intolerant like
Spark's, so the `CaseWhen` guard against NULL input arrays is no longer needed.

A floating-point map key stays a documented difference: Spark normalizes `-0.0`
to `+0.0` and canonicalizes `NaN` before storing a key, while the native
builders compare the raw Arrow values. `spark.comet.exec.strictFloatingPoint`
declines those key types.

Closes apache#4680

…d-dedup-policy

# Conflicts:
#	native/spark-expr/src/comet_scalar_funcs.rs
#	native/spark-expr/src/lib.rs
#	native/spark-expr/src/map_funcs/mod.rs

The upstream `datafusion-spark` map kernels read each row's entries at its own
offset but build the mask selecting the surviving keys from zero, then apply
that mask to the list's whole values array. Arrow's `filter` accepts a predicate
shorter than the array it filters, so on a sliced argument the mismatch silently
returns keys belonging to earlier rows rather than raising: keys `[[10], [20]]`
and values `[[100], [200]]`, both sliced to the second row, built `{10: 200}`
instead of `{20: 200}`. A `LIMIT` above a projection produces such an argument.

Compact any list argument whose values hold more than its offsets address
before validating or delegating, so the kernels see the layout they assume.
`map_from_entries` reached the same helper before this branch, so the bug is not
new to `map_from_arrays`; a fix belongs upstream as well.

Reported by @rich7420.

…d-dedup-policy

…d-dedup-policy

# Conflicts:
#	native/spark-expr/src/comet_scalar_funcs.rs

…the values expression

The previous commit restored `CASE WHEN keys IS NOT NULL AND values IS
NOT NULL` around `map_from_arrays`, and its regression test passed only
because the table held a single row. DataFusion's `AND` skips its right
side when the left side is false on every row of the batch, or on at
most a fifth of them; in a batch where most rows do have keys it
evaluates `values IS NOT NULL` on the whole batch, so a failing cast in
the values array still runs for the NULL-keys row and raises under ANSI
where Spark returns NULL.

Nest one `CaseWhen` per argument instead, as apache#5846 does. DataFusion
evaluates a THEN branch only on the rows its WHEN selected, so the values
expression is never evaluated for a row whose keys array is NULL, which
is what `BinaryExpression.eval` does.

Rewrite the regression test as a five-row table written in one partition
so every row shares a batch and most rows have keys. It fails with
`CAST_INVALID_INPUT` against the `AND` guard and passes with the nested
guards; the three `map_from_arrays` tests from apache#5846 pass as well.

`ArrayBasedMapBuilder` fixes a key's slot at its first occurrence and
only replaces its value, so `['a', 'b', 'a']` with `[1, 2, 3]` becomes
`{a -> 3, b -> 2}` rather than `{b -> 2, a -> 3}`. The datafusion-spark
kernels mirror that, but no test here told the two apart: the existing
LAST_WIN cases repeat one key, or repeat a key only in adjacent slots.

Add Rust unit tests for `map_from_arrays`, `map_from_entries` and
`str_to_map` that assert the keys and values in order, and fixture rows
for the same case. A map compares equal in any entry order, so the
fixtures pin the order through `map_keys` and `map_values`. Also cover a
NULL as the value that wins.

…aches the null guards

The nested null guards serialize each child a second time inside the
`map_from_arrays` call, so a stateful child advances independently in
each copy: the guard's copy sees every row while the constructor's copy
sees only the rows the guard selected. With

    map_from_arrays(IF(monotonically_increasing_id() % 2 != 0, array(1), NULL), array(2))

over sixteen rows in one partition, Spark returns eight maps and Comet
returned four (apache#5781). Under LAST_WIN this case used to fall back for
the policy alone, so running the policy natively exposed it there.

Port `NullGuardSupport` from apache#5867 unchanged in name, reason and
position, so that PR rebases by dropping the hunk, and decline a
nondeterministic child in `CometMapFromArrays.getSupportLevel` as
`Unsupported`; the projection falls back to Spark, which evaluates the
child once. apache#5867 still routes the same decline through the JVM codegen
dispatcher and applies it to `size`, `array_append` and `arrays_zip`.

Cover it with the query above as a Scala test on a one-partition table
under LAST_WIN, the same query in `map_from_arrays_dedup_policy.sql`,
and `map_from_arrays_nondeterministic_child.sql` for the default policy,
which mirrors the fixture in apache#5867 with `expect_fallback` in place of
`expect_dispatch`.

… native iterator

Spark's `ArrayBasedMapBuilder` reads `spark.sql.mapKeyDedupPolicy` when
an expression is first evaluated and keeps that builder, so a Dataset
executed again after the session setting changed still builds its maps
under the policy it started with. `CometExecIterator` forwarded the
setting from the task SQLConf on every new native iterator instead, so
the same Dataset switched behavior: executed under LAST_WIN, it raised
DUPLICATED_MAP_KEY on its next action once EXCEPTION was set, where
Spark kept returning the LAST_WIN map.

Carry the policy with the expression. New `MapFromArrays`,
`MapFromEntries` and `StrToMap` proto messages hold a
`map_key_dedup_policy` that the serde reads when it converts the plan,
which the Dataset reuses across actions, the way `Hour` carries its
timezone. The planner builds the native wrappers with that policy, and
they hand it to the datafusion-spark kernels through the session options
those kernels read, so whatever the session holds at execution time does
not apply. Drop the per-iterator forwarding and the session-level
`datafusion.spark.map_key_dedup_policy` setting, and the name-based
registrations the dedicated messages replace.

Cover it with a test that executes one Dataset twice across a policy
change, for all three constructors and in both directions, with Comet
disabled and enabled; it failed on the second `collect()` before this
change. A native unit test checks that the wrapper's policy wins over the
session option.

…p it

Spark's `ArrayBasedMapBuilder` is a lazy field of the map expression, so
`spark.sql.mapKeyDedupPolicy` is read the first time the expression is
evaluated and the expression keeps that builder afterwards. The previous
commit read the setting in the serde, which runs when the plan is
converted, and `explain()` converts a plan without evaluating anything.
A Dataset explained under EXCEPTION and then first collected under
LAST_WIN therefore raised DUPLICATED_MAP_KEY where Spark returns the
map, and the reverse direction returned a map where Spark raises.

Capture the policy in a `lazy val` on `CometNativeExec` instead. It is
forced by the first `doExecuteColumnar`, which `explain` does not reach,
and the node lives in the cached `executedPlan`, so every later action
reuses the value. The captured value reaches native through the plan's
config map, so `serializeCometSQLConfs` no longer reads the task SQLConf
for this setting; the shuffle-writer and write paths pass it the same
way.

That makes the proto messages the previous commit added unnecessary: the
problem was never where the policy travels but when it is read. Revert
them, along with the planner arms, the serde overrides and the wrappers'
constructor policy, so the constructors are wired as they were and the
kernels read the session option again.

Cover the new case with a test that materializes the plan under one
policy and first executes it under the other, in both directions and
with Comet disabled and enabled; it fails both ways without this change.

…d-dedup-policy

# Conflicts:
#	spark/src/main/scala/org/apache/comet/CometExecIterator.scala
#	spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala
#	spark/src/main/scala/org/apache/spark/sql/comet/operators.scala